A Practical Part - of - Speech TaggerDoug Cutting and Julian

We present an implementation of a part-of-speech tagger based on a hidden Markov model. The methodology enables robust and accurate tagging with few resource requirements. Only a lexicon and some unlabeled training text are required. Accuracy exceeds 96%. We describe implementation strategies and optimizations which result in high-speed operation. Three applications for tagging are described: phrase recognition; word sense disambiguation; and grammatical function assignment. 1 Desiderata Many words are ambiguous in their part of speech. For example, \tag" can be a noun or a verb. However, when a word appears in the context of other words, the ambiguity is often reduced: in \a tag is a part-of-speech label," the word \tag" can only be a noun. A part-of-speech tagger is a system that uses context to assign parts of speech to words. Automatic text tagging is an important rst step in discovering the linguistic structure of large text corpora. Part-of-speech information facilitates higher-level analysis, such as recognizing noun phrases and other patterns in text. For a tagger to function as a practical component in a language processing system, we believe that a tagger must be: Robust Text corpora contain ungrammatical constructions , isolated phrases (such as titles), and non-linguistic data (such as tables). Corpora are also likely to contain words that are unknown to the tagger. It is desirable that a tagger deal gracefully with these situations. EEcient If a tagger is to be used to analyze arbitrarily large corpora, it must be eecient|performing in time linear in the number of words tagged. Any training required should also be fast, enabling rapid turnaround with new corpora and new text genres. Accurate A tagger should attempt to assign the correct part-of-speech tag to every word encountered. Tunable A tagger should be able to take advantage of linguistic insights. One should be able to correct systematic errors by supplying appropriate a priori \hints." It should be possible to give diierent hints for diierent corpora. Reusable The eeort required to retarget a tagger to new corpora, new tagsets, and new languages should be minimal. 2 Methodology 2.1 Background Several diierent approaches have been used for building text taggers. Greene and Rubin used a rule-based approach in the TAGGIT program Greene and Rubin, 1971], which was an aid in tagging the Brown corpus Francis and Ku cera, 1982]. TAGGIT disambiguated 77% of the corpus ; the rest was done manually over a period of …